Repository navigation
HDDS-16359. Make CI survive Maven Central rate limiting - #11175
Conversation
CI jobs fail during dependency resolution, before anything is compiled, when Maven Central answers HTTP 429. The retry settings carried in MAVEN_OPTS since HDDS-5187 configure Wagon, which stopped being the resolver transport when the runner image moved to Maven 3.9, so they have had no effect for some time. Only the resolver's own default of three retries applies, giving up after roughly 30 seconds of a rate-limit window. Any pull request whose base predates a dependency bump on master restores a cache that no longer holds the pinned version, so it must fetch that artifact over the network and is exposed to this failure.
|
@rjgoyln looks like download problem still exists, please see https://github.com/apache/ozone/actions/runs/35516534110/job/106094445653#step:13:20 for example. |
|
Thanks for pointing this out. I went through the job log, and the retry setting is taking effect, but it is not enough to overcome the sustained 429s. The step spent 276.7s before failing. With I also found that the recurring issue is upstream of the retries: PRs that change For the retry settings, I think we overshot a bit. Sonatype recommends not retrying 429s more aggressively, and NiFi/Hudi use I’d suggest:
Ratis also adopted similar settings in RATIS-2698, so it may be worth keeping that in sync. WDYT? |
|
Thanks @rjgoyln, created HDDS-16648. Can you please also make the same change for Ratis (both |
Sure, I'll make the same change for both the |
Summary
A PR based on an older master can restore a Maven cache built from newer POMs, causing a missing dependency to be fetched from Maven Central. If Central returns HTTP 429, the job fails during dependency resolution before compilation or tests.
The existing retry settings use
maven.wagon.http.*, but Maven 3.9 resolves dependencies throughmaven-resolver-transport-http, which usesaether.connector.http.retryHandler.*. The Wagon properties are ignored, leaving only the resolver's default of three retries (~30s), matching the 32s failure seen in the JIRA job.This changes the settings to the resolver properties, increasing retry time to ~275s. The maximum individual wait remains 50s, below the resolver's 300s
intervalMax, and existing job timeouts still bound total runtime.maven.wagon.http.poolis removed for the same reason.http.keepAliveremains because it is JDK-level.JIRA: https://issues.apache.org/jira/browse/HDDS-16359
Tests
Tested with Maven 3.9.16 against a repository returning HTTP 429:
MAVEN_OPTSmaven.wagon.http.retryHandler.count=0aether.connector.http.retryHandler.count=0aether.connector.http.retryHandler.count=10The Wagon setting has no effect; the resolver setting works as expected.